Note on primitive permutation groups and a diophantine equation

If a group G has a maximal subgroup H, tl-.en G acts by right multiplication on the set fi of right cosets of H in G as a primitive, but not necessarily faithful, permutation group. As part of the search for new finite simple groups it is of interest to determine all possible groups G, and in particular all simple groups, which contain a given group H as a maximal subgroup. This problem is very difficult, but if the permutation representation induced on In has small rank and is faithful it has been solved for certain groups H. In this paper we assume that G is faithful on In and that H has an orbit A in c1 such that PSL(m, q)~ HA s PlYL(m, q) in its natural representation on the points or hyperplanes of the projective space, for some m ‘3 and prime power q. The case in which G has rank 2 on a was dealt with by H. Zassenhaus [l l] and D.R. Hughes [7]. We shall show that there exist no groups G which have rank 3 on a. This result is a generalisation of Bannai [2] which has the additional assumptions that H is faithful on A and q is even. Using a recent result of one of the authors [9] and the powerful restrictions on intersection numbers developed by D.G. HIgm~r. [4] we could show that either q = 3, m = 5, or q = 5 and 5”’ + 11 is a perfect square. The former case was eliminated by considering the action of a Sylow 13-subgroup. A very complicated general argument involving geometry and a Sylow p-subgroup, where p is a prime greater than 5 dividing (5”-* l), could be constructed to eliminate the other case. However the diophantine equation